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1.  Introduction 


1.1  Purpose  and  Overview 

The  U.S.  Army  Research  Laboratory  (ARL),  Human  Research  and  Engineering  Directorate 
(HRED)  conducted  a  human  factors  evaluation  of  the  OH-58F  Kiowa  Warrior  Cockpit  and 
Sensor  Upgrade  Program  (KW  CASUP)  crewstation  during  9-20  July  2012  at  the  Systems 
Simulation  and  Development  Directorate  (SSDD)  Apex  Laboratory,  Redstone  Arsenal,  AL.  The 
purpose  of  the  evaluation  was  to  identify  design  characteristics  of  the  crewstation  that  enhanced 
or  degraded  pilot  performance.  Aircrew  workload  and  situation  awareness,  the  crewstation 
interface,  pilot  visual  workload,  and  the  potential  for  pilot  simulator  sickness  were  assessed. 
Additionally,  workload  was  assessed  for  level  of  interoperability  2  (LOI  2)  tasks  that  the  aircrew 
perfonned  with  an  unmanned  aircraft  system  (UAS)  during  missions.  LOI  2  is  defined  as  the 
“direct  receipt  of  UAS  sensor  data  and  metadata”.  The  Human  Factors  Engineering  (HFE)  #3 
crewstation  assessment  was  the  third  in  a  series  of  crewstation  assessments  and  the  first  time  that 
LOI  2  with  a  UAS  was  evaluated. 

The  OH-58F  crewstation  simulator  (figures  1  and  2)  was  used  to  conduct  the  evaluation.  The 
human  factors  evaluation  is  part  of  the  continuous  assessment  process  to  develop  and  refine  the 
crewstation  design.  The  continuous  assessment  process  includes  modeling,  simulation, 
developmental  and  operational  testing. 

Pilots  received  three  days  of  training  prior  to  the  beginning  of  the  evaluation.  The  training  was 
conducted  at  SSDD  and  consisted  of  classroom  instruction  and  hands-on  flight  training  using  a 
desktop  simulator  and  the  OH-58F  crewstation  simulator. 

The  aircrews  flew  similar  missions  during  training  that  they  later  flew  during  the  record  trials. 
The  mission  scenarios  were  based  on  a  battlefield  environment  simulating  southwest  Asia.  Each 
successive  mission  increased  in  difficulty  in  order  to  impose  progressively  greater  workload  on 
the  pilots.  The  aircrews  performed  route,  area,  and  landing  zone/pick-up  zone  reconnaissance, 
call-for-fire,  and  specific  Aircrew  Training  Manual  (ATM)  tasks  with  a  UAS  during  missions. 
Each  ATM  task  has  prescribed  conditions  and  standards  to  which  both  crewmembers  had  to 
perform  to  ensure  mission  accomplishment.  The  pilots  rotated  seat  positions  during  the  assessment. 

During  the  fonnal  evaluation,  three  sets  of  aircrews  conducted  three  reconnaissance  missions 
(for  a  total  of  nine  missions).  The  missions  consisted  of  flight  segments  in  visual  meteorological 
conditions  (VMC),  instrument  meteorological  conditions  (IMC),  and  tactical  conditions.  The 
mission  scenarios  were  developed  by  the  Training  and  Doctrine  Command  (TRADOC) 
Capability  Manager,  Reconnaissance  Attack  (TCM  RA)  office,  Fort  Rucker,  AL.  The  scenarios 
were  developed  in  accordance  with  scout  aircraft  tactics,  techniques,  and  procedures  (TTP). 
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Prior  to  each  mission,  the  pilots  received  a  briefing  about  mission  goals;  at  the  conclusion  of 
each  mission,  they  completed  human  factors  surveys.  Before  and  after  each  mission,  they 
completed  the  Simulator  Sickness  Questionnaire  (SSQ);  after  each  mission,  they  completed  the 
Bedford  Workload  Rating  Scale  (BWRS),  and  China  Lake  Situational  Awareness  (CLSA)  rating 
scale.  The  pilots  completed  the  Pilot-Crewstation  Interface  (PCI)  questionnaire  after  they 
completed  all  of  their  missions.  During  each  mission,  pilots  wore  an  eye  tracker  used  to  assess 
their  visual  workload.  In  addition  to  the  pilot  data,  subject  matter  experts  (SMEs)  provided  an 
independent  assessment  of  aircrew  workload,  situation  awareness,  and  mission  success.  After 
each  mission,  the  SMEs  completed  an  aircrew  workload,  situation  awareness,  and  mission 
success  survey.  After  the  aircrews  completed  the  mission  and  surveys,  they  participated  in  a 
mission  debriefing  and  after  action  review  (AAR). 


Figure  1.  OFI-58F  crewstation  simulator. 
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Figure  2.  OFI-58F  instrument  panel. 

1.2  Assessment  of  Crew  Workload 

A  common  definition  of  pilot  workload  is  .  .  the  integrated  mental  and  physical  effort  required 
to  satisfy  the  perceived  demands  of  a  specified  flight  task”  (Roscoe,  1985).  It  is  important  to 
assess  pilot  workload  because  mission  accomplishment  is  related  to  the  mental  and  physical 
ability  of  the  crew  to  effectively  perform  their  flight  and  mission  tasks.  If  one  or  both  pilots 
experience  excessively  high  workload  while  performing  flight  and  mission  tasks,  the  tasks  may 
be  performed  ineffectively  or  abandoned.  In  order  to  assess  whether  the  pilots  are  task- 
overloaded  during  the  mission  profiles,  the  level  of  workload  for  each  pilot  must  be  evaluated. 

1.3  Bedford  Workload  Rating  Scale  (BWRS) 

The  pilots  completed  the  BWRS  (appendix  A)  immediately  after  each  mission  to  rate  the  level  of 
workload  that  they  experienced  when  performing  ATM  tasks  during  missions.  The  ATM  tasks 
(appendix  A)  were  selected  by  personnel  from  the  TCM  RA,  ARL  HRED,  SSDD  and  the  Armed 
Scout  Helicopter  Program  Managers  Office  because  they  were  estimated  to  have  the  most  impact 
on  pilot  workload  during  the  missions.  SMEs  completed  the  BWRS  immediately  after  each 
mission  to  rate  overall  workload  for  each  pilot.  SME  workload  comments  are  listed  in 
appendix  B. 
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The  BWRS  has  been  used  extensively  by  the  military,  civil,  and  commercial  aviation 
communities  for  pilot  workload  estimation  (Roscoe  and  Ellis,  1990).  It  requires  pilots  to  rate  the 
level  of  workload  associated  with  a  task  based  on  the  amount  of  spare  capacity  they  feel  they 
have  to  perform  additional  tasks.  Spare  workload  capacity  is  an  important  commodity  for  scout 
pilots  because  they  are  often  required  to  perform  several  tasks  concurrently.  For  example,  pilots 
often  perform  navigation  tasks,  communicate  via  multiple  radios,  monitor  aircraft  systems,  and 
assist  the  pilot  on  the  controls  with  flight  tasks  (e.g.,  maintain  airspace  surveillance)  within  the 
same  time  interval.  Mission  perfonnance  is  reduced  if  pilots  are  task  saturated  and  have  little  or 
no  spare  capacity  to  perform  other  tasks.  Integration  of  the  OH-58F  crewstation  should  help 
ensure  that  pilots  can  maintain  adequate  spare  workload  capacity  while  perfonning  flight  and 
mission  tasks.  The  OH-58F  has  a  Capability  Development  Document  (CDD)  requirement  that 
aircrew  workload  not  exceed  6.0  (Threshold)  and  5.0  (Objective)  on  the  BWRS. 

1.4  Visual  Workload 

An  eye  tracker  was  used  during  the  evaluation  to  assess  visual  gaze  and  dwell  times  for  the 
pilots.  The  data  were  collected  to  help  detennine  how  well  the  design  of  the  crewstation  allowed 
the  flying  pilot  to  remain  focused  outside  the  aircraft  during  visual  flight  rules  (VFR)  flight  and 
how  well  the  non-flying  pilot  was  able  to  maintain  visual  focus  outside  the  aircraft  to  assist  with 
navigation  (e.g.,  identification  of  terrain  features),  local  security,  terrain  flight,  etc.  Visual  gaze 
and  dwell  times  also  help  identify  if  pilots  experience  excessive  visual  workload  or  cognitive 
capture  because  they  had  problems  interpreting  information  presented  to  them  on  the  crewstation 
displays. 

1.5  Assessment  of  Crew  Situational  Awareness  (SA) 

SA  can  be  defined  as  the  pilot’s  mental  model  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  formal  definition  is  “the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  It  was  important  to  assess  SA  because  it  had  a 
direct  impact  on  pilot  and  system  perfonnance.  Good  SA  should  increase  the  probability  of  good 
decision-making  and  perfonnance  by  aircrews  when  conducting  flight  and  mission  tasks  in  the 
OH-58F. 

1.6  China  Lake  Situational  Awareness  (CLSA)  Scale 

The  CLSA  (appendix  C)  is  a  unidimensional  rating  scale  for  pilots  to  report  their  perceived  SA. 
The  CLSA  uses  a  five-point  scale  that  requires  pilots  to  rate  their  knowledge  of  aircraft  energy 
state,  tactical  environment  and  mission,  ability  to  anticipate  and  accommodate  trends,  and  if  they 
shed  tasks  during  the  mission. 
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1.7  Assessment  of  the  Pilot-Crewstation  Interface  (PCI) 

The  PCI  directly  affects  aircrew  workload  and  SA  during  a  mission.  A  crewstation  that  is 
designed  to  augment  the  cognitive  and  physical  abilities  of  crews  will  minimize  workload, 
enhance  SA,  and  contribute  to  successful  mission  performance.  The  pilots  completed  a  PCI 
questionnaire  (appendix  D)  to  identify  any  problems  with  the  usability  of  the  controls,  displays, 
or  subsystems. 

1.8  Assessment  of  Simulator  Sickness 

Simulator  sickness  has  been  defined  as  a  condition  where  pilots  suffer  physiological  discomfort 
in  the  simulator,  but  not  while  flying  the  actual  aircraft  (Kennedy  et  ah,  1989).  It  is  generally 
believed  that  simulator  sickness  is  caused  by  a  mismatch  either  between  the  visual  and  vestibular 
sources  of  infonnation  about  self-motion,  or  between  the  sensory  infonnation  (e.g.,  acceleration 
cues)  presented  by  the  simulator  and  the  sensory  information  presented  by  the  primary  aircraft 
that  the  pilot  operates.  When  the  sensory  infonnation  presented  by  the  simulator  does  not  match 
the  aircraft,  the  pilot’s  nervous  system  reacts  adversely  to  the  sensory  mismatch  and  the  pilot 
begins  to  experience  discomfort.  Characteristics  of  simulator  sickness  include  nausea,  dizziness, 
drowsiness  and  several  other  symptoms  (Kennedy  et  ah,  1989).  It  is  important  to  assess  simulator 
sickness  because  the  discomfort  felt  by  pilots  can  be  distracting.  Pilot  distraction  is  one  of  the 
operational  consequences  of  simulator  sickness  listed  by  Crowley  (1987).  If  pilots  are  distracted  by 
the  discomfort  they  feel  during  missions,  their  perfonnance  is  likely  to  suffer.  Additionally,  the 
discomfort  could  influence  the  perceived  levels  of  workload  and  SA  that  the  pilots  experienced 
during  a  mission. 

1.9  Simulator  Sickness  Questionnaire  (SSQ) 

The  SSQ  was  administered  to  the  pilots  to  estimate  the  severity  of  physiological  discomfort  that 
they  experienced  during  missions  and  help  assess  whether  they  were  being  distracted  by  the 
discomfort.  The  SSQ  (Kennedy  et  ah,  1993)  is  a  checklist  of  16  symptoms.  The  16  symptoms  are 
categorized  into  three  subscales.  The  subscales  are  Oculomotor  (e.g.,  eyestrain,  difficulty 
focusing,  blurred  vision),  Disorientation  (e.g.,  dizziness,  vertigo),  and  Nausea  (e.g.,  nausea, 
increased  salivation,  burping).  The  three  subscales  are  combined  to  produce  a  Total  Severity 
score.  The  Total  Severity  score  is  an  indicator  of  the  overall  discomfort  that  the  pilots 
experienced  during  the  mission. 

1.10  Subject  Matter  Experts  (SMEs) 

Three  SMEs  typically  observed  the  missions  and  rated  crew  workload,  crew  SA,  and  mission 
success.  The  SMEs  provided  an  independent  assessment  of  the  workload  and  SA  levels 
experienced  by  the  crews.  They  also  helped  identify  whether  problems  with  crew  workload  or 
crew  SA  contributed  to  lack  of  mission  success. 
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The  SMEs  were  TCM  RA  personnel  who  had  substantial  experience  conducting  armed 
reconnaissance  missions  and  were  familiar  with  the  OH-58F  crewstation.  They  observed  each 
mission  from  the  Battlemaster  station  where  they  could  monitor  crewstation  displays  and  the  out- 
the-window  (OTW)  view  provided  to  the  crew.  They  also  listened  to  all  audio  communications 
between  crewmembers  and  outside  sources  during  the  missions. 

An  additional  SME  was  used  during  the  crewstation  assessment  to  perfonn  the  role  of  Ground 
Control  Station  operator.  The  SME  was  from  TCM  Unmanned  Aircraft  Systems.  He  controlled 
the  UAS  aircraft  and  payload  and  conducted  sensor  scans  for  the  pilots. 

1.11  Simulation  Environment 

SSDD  provides  modeling  and  simulation  support  of  weapon  systems  early  in  the  acquisition 
process.  This  is  accomplished  through  several  methods,  including  man-in-the-loop  simulators, 
distributed  simulation  experimentation,  and  constructive  simulation  development  in  the  SSDD 
Apex  Laboratory. 

The  Advanced  Prototype  Engineering  and  Experimentation  (APEX)  Lab  is  High-Level 
Architecture  (HLA)  and  Distributed  Interactive  Simulation  (DIS)  compliant,  and  has  the 
capability  to  connect  to  the  Anny’s  Battle  Labs  and  other  distributed  simulation  facilities 
through  the  Defense  Research  and  Engineering  Network  (DREN). 

The  APEX  Lab  includes  a  Battlemaster  control  center  that  has  access  to  each  simulation  playing 
on  the  network  by  means  of  a  One  Semi-Automated  Forces  (OneSAF)  test  bed  tenninal,  data 
collection  devices,  headset  communications,  and  video  monitoring.  All  exercises  are  controlled 
from  the  Battlemaster  station  to  ensure  that  all  players  are  engaged  in  the  exercise  and  all  data 
collection  devices  are  active.  Time  coordination  and  time  stamping  of  video  collection  devices  is 
achieved  through  an  integrated  Global  Positioning  System  (GPS)  clock.  Audio  and  video  are 
captured  and  routed  throughout  the  lab  and  various  conference  rooms  through  a  custom  video 
capture  and  switching  system. 

The  APEX  Lab  has  a  complete  synthetic  environment  development  team  that  is  able  to  develop 
custom,  correlated  terrain  databases  that  are  designed  to  specifically  enhance  realism  of  the 
immersive  environment  and  support  operational  scenarios  for  each  event.  The  Battlefield  Highly 
Immersive  Visual  Environment  (BHIVE)  provides  this  immersiveness  with  high-fidelity  OTW 
terrain  databases  and  image  generators.  The  BHIVE  was  developed  in  support  of  weapon  system 
evaluation  in  an  HLA/DIS  compliant,  man-in-the-loop,  virtual  environment.  It  was  designed  with 
a  roll-in/roll-out  capability  to  allow  integration  of  several  types  of  devices  into  the  environment 
through  a  standard  interface.  This  provides  the  flexibility  to  immerse  multiple  types  of  cockpits 
in  a  realistic  and  reusable  synthetic  world.  Six  projectors  are  used  to  project  the  OTW  view  onto 
an  180°  x  60°  directional  curved  dome.  The  projection  system  is  capable  of  edge  blending  for 
high-definition  synthetic  environments. 
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2.  Method 


2.1  Pilots 

Six  pilots  participated  in  the  crewstation  assessment.  The  pilots  were  warrant  officers  and  rated 
in  the  OH-58D.  Three  pilots  held  the  rank  of  CW2  and  three  pilots  held  the  rank  of  CW4.  They 
were  from  the  2-6  CAV  (25ID),  7-17  CAV  (159  CAB),  1  -  13th  AVN  (Directorate  of  Evaluation 
and  Standardization),  2-17  CAV,  and  2-6  CAV.  The  pilots  represented  a  broad  range  of 
experience  with  their  total  flight  hours  ranging  from  500-4200  h.  The  relevant  demographic 
characteristics  of  the  pilots  are  listed  in  table  1 . 

Table  1.  Pilot  demographics  (N  =  6). 


Summary  of 
Demographic 
Characteristics 

Age 

(years) 

Flight  Hours 
in  OH-58D 

Total  Flight  Hours 
in  Army  Aircraft 

Mean 

33.0 

2121 

2232 

Median 

32.5 

2088 

2145 

Range 

26-42 

400-3850 

500^200 

2.2  Data  Collection 

The  pilot  questionnaires  were  developed  in  accordance  with  published  guidelines  for  proper 
format  and  content  (O’Brien  and  Charlton,  1996).  A  pretest  was  conducted  to  refine  the 
questionnaires  and  to  ensure  that  they  could  be  easily  understood  and  completed  by  pilots  and 
SMEs. 

The  pilots  completed  the  workload  and  SA  questionnaires  after  each  mission.  They  completed 
the  SSQ  before  and  after  each  mission.  The  pilots  completed  the  PCI  questionnaire  after  they 
completed  all  of  their  missions.  The  SMEs  completed  questionnaires  after  each  mission. 
Additional  data  were  obtained  from  the  pilots  and  the  SME  members  during  post-mission 
discussions  and  the  final  AAR.  Questionnaire  results  were  clarified  with  information  obtained 
during  post-mission  discussions  and  the  final  AAR. 

2.3  Eye  Tracker  System 

Pilot  visual  gaze  and  dwell  times  were  collected  with  an  eye  tracking  system  (Model  501)  from 
Applied  Science  Laboratories  (ASL)  and  a  head  tracking  system  (Polaris  Spectra)  from  Northern 
Digital  Incorporated  (NDI).  These  systems  were  used  because  they  allowed  unrestricted  head 
movement  during  data  collection  and  were  compatible  with  the  HGU-56  flight  helmet  worn  by 
the  pilots.  This  technology  allowed  the  collection  of  digital  data  that  specified  point  of  gaze  with 
respect  to  stationary  objects  within  the  crewstation.  The  ASL  software  allowed  data  collectors  to 
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continuously  monitor  the  eye  position  of  the  pilots  by  crosshairs  superimposed  over  live  imagery 
(figure  3).  The  software  also  included  a  built-in  analysis  tool  that  allowed  data  to  be  viewed  in 
tabular  or  graphical  format. 


Figure  3.  Eye  tracker  scene  camera  monitors  and  control  panel  interface. 

2.4  Data  Analysis 

Pilot  responses  to  the  BWRS,  CLSA,  SSQ,  and  PCI  questionnaires  were  analyzed  with  means 
and  percentages.  The  eye  tracker  data  were  summarized  by  calculating  the  total  percentage  of 
fixations  that  occurred  for  the  different  areas  of  interest  (AOI).  Eight  AOIs  were  created  for  the 
pilot  and  copilot:  5><7  multifunctional  display  (MFD),  6  x  8  MFD,  kneeboard  (both  pilots), 
OTW,  lower  console,  and  outer  left  instrument  panel  (copilot).  A  final  category,  called  “Other,” 
captured  eye  fixations  not  focused  on  a  specific  AOI.  Results  of  the  eye  tracker  data  are  listed  in 
figure  4. 

2.5  Evaluation  Limitations 

Limitations  included  the  small  sample  size  of  pilots  (N=6)  who  participated  in  the  crewstation 
simulation  assessment,  limited  amount  of  training  provided  to  the  pilots,  and  hardware/software 
limitations.  The  primary  hardware  and  software  limitations  are  listed  below: 

•  Built-In  Test  (BIT)  functions  were  not  available 

•  Hands-On  Grip  (HOG)  functional  for  Joint  Variable  Message  Format  (JVMF)  free  text 
only 

•  Video  Tape  Recorder  (VTR)  was  nonfunctional 
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•  Health  Usage  Monitoring  System  (HUMS)  was  not  available 

•  Display  map  did  not  have  Controlled  Image  Base  (CIB)  and  12.5  software  functionality 

•  Pulse  Interval  Module  (PIM)  codes  were  not  available 

•  Satellite  Demand  Assigned  Multiple  Access  (DAMA)  and  frequency  hop  were 
nonfunctional 

•  Video  rocker  switches  were  nonfunctional 

•  Heim  re  was  functional  in  lock-on-before-launch  (LOBL)  mode  only 

•  Certain  display  font  sizes  were  slightly  inaccurate 

•  Video  rocker  switches  were  nonfunctional 

•  Collective  throttle  was  nonfunctional 

•  Feature  track  is  the  only  mode  that  will  allow  tracking  a  building 

•  Stability  Control  Augmentation  System  (SCAS)  was  nonfunctional 

•  Circuit  breaker  panels  were  nonfunctional  and  not  representative  of  final  design 

•  Nose-mounted  sensor  (NMS)  Linear  Motion  Control  did  not  function  if  the  NMS  was 
pointed  off  the  ground 

These  limitations  are  common  because  of  funding  and  time  constraints  as  well  as  replicating  a 
complex  aviation  system  in  a  simulator.  However,  the  information  and  data  listed  in  the  Results 
and  Summary  sections  of  this  report  should  be  interpreted  based  on  these  limitations.  Additional 
data  should  be  collected  during  future  simulations  and  tests  to  augment  and  expand  the  findings 
contained  in  this  report. 


3.  Results 


3.1  Crew  Workload 

3.1.1  Mean  Workload  Ratings  For  Flight  and  Mission  Tasks 

The  average  mission  workload  rating  was  2.56  for  the  pilot  and  3.44  for  the  copilot  (figure  5). 
These  ratings  indicate  that  the  pilots  and  copilots  typically  felt  that  workload  was  tolerable  for  all 
flight  and  mission  tasks  they  perfonned  during  the  missions.  Additionally,  the  ratings  indicate 
that  the  pilots  and  copilots  typically  felt  they  had  enough  spare  workload  capacity  to  perform  all 
desirable  additional  tasks  (within  the  same  time  interval)  during  missions.  The  average  workload 
ratings  for  LOI  2  with  the  UAS  were  2.67  for  the  pilot  and  3.00  for  the  copilot.  Based  on  these 
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ratings  and  post-mission  discussions  with  the  aircrew,  the  addition  of  UAS  LOI  2  tasks  do  not 
appear  to  significantly  increase  workload  for  the  pilot  and  copilot  during  missions. 

The  average  workload  ratings  for  flight  and  mission  tasks  (appendix  A)  provided  by  the  pilots 
and  copilots  were  lower  than  the  Objective  (5.0)  and  Threshold  (6.0)  BWRS  workload  rating 
requirements  contained  in  the  OH-58F  CDD  except  for  the  task  “Perform  an  Autorotation.” 

During  one  mission,  the  copilot  helped  perform  an  autorotation  and  provided  a  workload  rating 
of  “5”  for  the  task.  He  commented  that  the  higher  workload  he  experienced  during  the 
autorotation  was  due  to  the  intensive  nature  of  the  task  and  that  no  additional  tasks  could  be 
performed  in  order  to  safely  land  the  aircraft. 

3.1.2  SME  Workload  Ratings 

SMEs  provided  an  overall  Bedford  workload  rating  for  each  pilot  during  each  mission  that  they 
observed.  The  average  SME  Bedford  workload  rating  (figure  4)  was  3.26  for  the  pilot  and  4.48 
for  the  copilot  for  all  missions.  These  ratings  indicate  that  the  SMEs  believed  that  workload  was 
tolerable  for  the  pilots  and  copilots  with  insufficient  spare  capacity  for  easy  attention  to 
additional  tasks  for  the  copilots.  The  ratings  are  lower  than  the  Objective  (5.0)  and  Threshold 
(6.0)  BWRS  workload  rating  requirements  contained  in  the  OH-58F  CDD. 


Figure  4.  Pilot  and  copilot  visual  gaze  and  dwell  times. 
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Aircrew  and  SME  Workload  Ratings 


Aircrew  Ratings  SME  Ratings 


Figure  5.  Aircrew  and  SME  workload  ratings. 

3.1.3  Workload  Comparison  for  HFE  #1,  HFE  #2,  and  HFE  #3  Crewstation  Assessments 

The  average  overall  aircrew  and  SME  workload  ratings  for  the  three  crewstation  simulation 
assessments  (HFE  #1,  HFE  #2,  and  HFE  #3)  are  listed  below  (table  2).  These  ratings  indicate 
that  the  pilots,  copilots,  and  SMEs  typically  felt  that  workload  was  tolerable  for  all  flight  and 
mission  tasks  that  were  performed  during  the  missions.  Additionally,  the  ratings  indicate  that  the 
pilots  and  copilots  typically  felt  they  had  enough  spare  workload  capacity  to  perform  all 
desirable  additional  tasks  during  missions.  The  SME  workload  ratings  indicate  that  they  felt  that 
the  copilots  often  had  insufficient  spare  capacity  for  easy  attention  to  additional  tasks.  The 
workload  ratings  for  HFE  #1,  HFE  #2,  and  HFE  #3  were  lower  than  the  Objective  (5.0)  and 
Threshold  (6.0)  BWRS  workload  rating  requirements  contained  in  the  OH-58F  CDD. 


Table  2.  Average  workload  ratings  for  crewstation  assessments. 


Aircrew  and  SME 
Workload  Ratings 

HFE  #1 

HFE  #2 

HFE  #3 

Aircrew  Workload  Ratings 

Pilot -3.0 

Copilot  -  3.17 

Pilot -2.75 

Copilot  -  3.38 

Pilot -2.56 

Copilot  -  3.44 

SME  Workload  Ratings 

Pilot -3.88 

Copilot  -  4.00 

Pilot -3.31 

Copilot  -  3.46 

Pilot -3.26 

Copilot  -  4.48 

CDD  Workload  Requirement 

“Combined  mission  workload  tasks  not  to 
exceed  Bedford  workload  rating  of  5.0  (Objective)  and  6.0  (Threshold)” 
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3.1.4  Visual  Workload 


Figure  5  shows  the  percentage  of  time  that  the  pilots  were  visually  focused  (during  VFR  flight) 
on  each  AOI  during  the  missions.  It  is  interesting  to  note  that  the  copilots  typically  spent  only 
4%  of  the  time  visually  focused  OTW  during  missions.  The  copilot  needs  to  periodically 
maintain  visual  focus  outside  the  aircraft  to  assist  with  navigation  (e.g.,  identification  of  terrain 
and  cultural  features)  and  airspace  surveillance.  The  percentage  of  time  that  the  copilot 
maintained  visual  focus  outside  the  aircraft  was  about  the  same  as  it  was  during  the  HFE  #1 
crewstation  assessment  (table  3).  Maintaining  visual  focus  outside  the  aircraft  for  only  4%  of  a 
typical  zone  reconnaissance  mission  is  likely  too  low  a  percentage  of  time  to  adequately  assist 
the  pilot  with  crew  tasks  such  as  obstacle  avoidance  and  terrain  flight  navigation.  The  low 
percentage  of  time  that  the  copilots  were  visually  focused  outside  the  aircraft  was  mostly  due  to 
the  workload  required  to  manage  information  on  the  crewstation  displays,  operate  the  NMS  and 
the  lack  of  in-depth  experience  that  the  copilots  had  with  the  crewstation  interface.  It  will  be 
important  to  assess  visual  gaze  during  the  Limited  User  Test  and  Initial  Operational  Test  and 
Evaluation  to  detennine  if  copilot  workload  precludes  maintaining  adequate  visual  focus  outside 
the  aircraft. 

Table  3.  Comparison  of  eye  tracker  results  for  OH-58F,  AH-64D  and  ARH  simulations. 


AH-64D/UAS 
Workload 
Assessment 
(Block  III) 

AH-64D 
Workload 
Assessment 
(Block  III) 

ARHa  HFE- 
CAASb 
Evaluation 

OH-58F  HFE  #1 
Evaluation 

OH-58F  HFE  #2 
Evaluation 

OH-58F  HFE  #3 
Evaluation 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Flying 

Pilot 

(%) 

Non- 

Flying 

Pilot 

(%) 

Outside 

75 

4 

75 

6 

75 

3 

61 

7 

50 

14 

77 

4 

Inside 

25 

96 

25 

94 

25 

97 

39 

93 

50 

86 

23 

96 

‘‘Armed  reconnaissance  helicopter.  bCommon  Aviation  Architecture  System. 


The  pilots  typically  spent  77%  of  the  time  visually  focused  OTW  during  VFR  flight.  The  amount 
of  time  (77%)  that  the  pilots  were  visually  focused  outside  the  aircraft  was  higher  than  in  the 
HFE  #1  and  HFE  #2  crewstation  assessments. 

3.1.5  Comparison  of  Eye  Tracker  Data 

Table  3  shows  a  comparison  of  OH-58F,  AH-64D,  and  ARH  eye  tracker  data  for  VFR  flight 
during  simulations.  While  the  simulator,  missions,  and  personnel  experience  levels  were 
different  for  each  simulation  evaluation,  it  is  interesting  to  note  that  the  flying  pilot  maintained 
visual  gaze  and  dwell  times  outside  of  the  crewstation  for  50%  to  77%  of  the  time  and  the  non¬ 
flying  pilot  for  4%  to  14%  of  the  time  during  missions. 
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3.2  Crew  SA 


3.2.1  Pilot  and  Copilot  SA  Ratings 

The  pilots  and  copilots  reported  that  they  typically  experienced  good  SA  during  missions,  were 
able  to  maintain  knowledge  of  the  aircraft  energy  state,  tactical  environment  and  mission,  were 
able  to  partially  anticipate  and  accommodate  trends,  and  had  minimal  task  shedding  (due  to  high 
workload)  during  the  missions.  The  pilots  and  copilots  typically  reported  “fairly  high”  to  “very 
high”  levels  of  SA  for  the  location  of  their  ownship  during  missions,  route  infonnation  (e.g., 
phase  lines),  location  of  friendly  and  enemy  units,  cultural  features,  and  status  of  the  aircraft 
systems  (e.g.,  fuel  consumption).  Table  4  compares  SA  ratings  for  the  HFE  #1,  HFE  #2,  and 
HFE  #3  crewstation  assessments.  For  all  assessments,  the  pilots  and  copilots  reported  that  they 
typically  experienced  good  SA  during  missions.  The  CLSA  rating  scale  is  on  page  32  for 
reference. 


Table  4.  Aircrew  SA  ratings  for  crewstation  assessments. 


HFE  #1 

HFE  #2 

HFE  #3 

Pilot -2.55 

Pilot -2.50 

Pilot  -  1.67 

Copilot  -  2.33 

Copilot  -  2.75 

Copilot  -  2.22 

3.2.2  SME  SA  Ratings 

The  SMEs  provided  an  independent  assessment  of  SA  based  on  the  scale  shown  in  table  5.  The 
mean  SME  SA  rating  was  2.85.  This  indicates  that  the  SMEs  believed  that  the  crews  typically 
had  adequate  levels  of  S  A  with  some  variation  between  aircrew  perception  of  entities  on  the 
battlefield  and  reality.  During  several  mission  segments,  some  crews  or  individual  crewmembers 
had  reduced  SA.  The  SMEs  made  several  comments  that  crew  fixation  on  the  NMS  imagery  and 
maps  displayed  on  the  MFDs,  lack  of  experience  using  a  ETAS  to  perfonn  reconnaissance  tasks, 
low  experience  levels  of  young  pilots,  and  periods  of  increased  workload  reduced  overall  aircrew 
SA  during  several  mission  segments.  Table  6  compares  SA  ratings  for  the  HFE  #1,  HFE  #2,  and 
HFE  #3  crewstation  assessments.  The  ratings  indicate  that  the  SMEs  believed  that  the  crews 
typically  had  adequate  levels  of  SA  with  some  variation  between  aircrew  perception  of  entities 
on  the  battlefield  and  reality  during  the  three  assessments. 
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Table  5.  SME  SA  rating. 


SME  SA  Ratings 

1 .  Crew  was  consistently  aware  of  all  entities  on  the  battlefield. 

^  Crew  was  aware  of  the  battlefield  with  minor  or  insignificant  variation 

between  perception  and  reality. 

◄ - 

^  Crew  was  aware  of  the  battlefield.  Variation  between  reality  and  perception 
did  not  significantly  impact  mission  success. 

^  SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the 
success  of  the  mission. 

5.  Lack  of  SA  caused  mission  failure. 


Mean  Rating 


2.85 

(SD=  1.16) 


Table  6.  SME  SA  ratings  for  crewstation  assessments. 


HFE  #1 

HFE  #2 

HFE  #3 

2.77 

2.54 

2.85 

3.3  SME  Mission  Success  Ratings 

At  the  end  of  each  mission,  SMEs  rated  whether  the  mission  was  a  success  or  failure.  The 
criteria  that  the  SME  used  to  rate  mission  success  or  failure  was  whether  the  aircrew  completed 
the  mission  requirements  and  did  not  get  shot  down  or  crash.  The  SMEs  rated  all  of  the  missions 
(100%)  as  “successful”  and  “objectives  completed”  (figure  6). 

SME  Ratings  of  Mission  Success  &  Objectives 


Successful  Missions  Objectives  Completed 

Figure  6.  SME  ratings  of  mission  success  and  objectives. 
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3.4  Pilot-Crewstation  Interface  (PCI) 


The  pilots  were  mostly  favorable  in  their  ratings  of  the  crewstation  interface  (appendix  D).  They 
reported  they  were  able  to  effectively  use  the  MFD  pages  and  functions,  quickly  navigate 
through  the  pages,  subpages  and  overlays  on  the  crewstation  displays,  easily  use  the  switches  on 
the  cyclic  and  collective  and  easily  use  the  switches  to  control  the  NMS.  They  also  reported  that 
it  was  easy  to  detect  the  Warnings,  Cautions,  and  Advisories  on  the  MFD  and  entry  into 
operational  limits.  The  pilots  reported  that  the  control  display  system  (CDS)  5  software  was 
quicker  and  easier  to  use  compared  to  the  CDS  4  software.  They  also  reported  that  the  6  x  8 
display  enhanced  SA  and  that  NMS  functionality  is  an  improvement  versus  the  mast-mounted 
sight  (MMS)  on  the  OH-58D  Kiowa  Warrior.  The  pilots  reported  several  minor  crewstation 
improvements  that  should  be  made  to  the  OH-58F  (see  appendix  E). 

3.5  Simulator  Sickness 

The  pilots  and  copilots  reported  that  they  typically  experienced  very  mild  simulator  sickness 
symptoms  during  the  evaluation.  The  overall  mean  Total  Severity  (TS)  score  (post  mission)  for 
the  pilots  and  copilots  was  7.48  (table  7).  The  mean  TS  score  for  the  pilot  was  6.23  and  the  mean 
TS  score  for  the  copilot  was  8.73.  When  flying  the  aircraft,  the  pilots  were  visually  immersed  in 
the  changing  scene  outside  the  aircraft  and  often  transitioned  their  visual  gaze  inside  the  aircraft 
to  monitor  information  displayed  on  the  MFDs.  The  copilots  primarily  maintained  their  visual 
gaze  inside  the  aircraft  to  monitor  and  input  data  into  their  MFDs.  All  pilots  commented  that  the 
simulator  sickness  symptoms  were  typically  very  mild.  Overall,  the  simulator  did  not  appear  to 
induce  debilitating  simulator  sickness  symptoms  and  should  continue  to  be  a  suitable  simulation 
environment  for  future  training  and  assessments. 

Table  7.  Simulator  SSQ  ratings. 


Condition 

Nausea 

Subscale 

Oculomotor 

Subscale 

Disorientation 

Subscale 

Total  Severity 
Score  (Mean) 

Premission  pilot 

1.06 

1.68 

0 

1.25 

Premission  copilot 

1.06 

2.53 

0 

1.66 

Post  mission  pilot 

5.30 

5.90 

4.64 

6.23 

Post  mission  copilot 

10.60 

8.42 

1.55 

8.73 

Post  mission  combined 
(pilot  and  copilot) 

7.95 

7.16 

3.09 

7.48 

3.5.1  Comparison  of  OH-58F  Simulator  SSQ  Scores  to  Other  Helicopter  Simulators 

To  assess  whether  the  SSQ  ratings  provided  by  the  pilots  during  the  HFE  assessment  were 
similar  or  different  to  ratings  obtained  in  other  helicopter  simulators,  the  mean  SSQ  scores  for 
the  OH-58F  simulator  were  compared  to  the  mean  SSQ  scores  for  several  other  helicopter 
simulators  (table  8).  The  other  helicopter  simulators  were  the  ARH,  OH-58F  (HFE  #1  and 
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HFE  #2),  S-3H,  Sikorsky  RAH-66  Engineering  Development  Simulator  (EDS),  RAH-66 
Comanche  Portable  Cockpit  (CPC),  the  simulator  used  during  the  UH-60M  from  the  Early  User 
Demonstration  (EUD)  and  Limited  Early  User  Evaluation  (LEUE),  AH-64D,  CH-46E,  and  CH- 
53F.  These  simulators  typically  induced  very  mild  to  mild  simulator  sickness  symptoms  in  pilots 

Table  8.  Comparison  ofOH-58F  simulator  SSQ  ratings  with  other  helicopter  simulators. 


Simulator 

Nausea 

Oculomotor 

Disorientation 

Total  Severity  Score 

Subscale 

Subscale 

Subscale 

(Mean) 

ARH  Simulator 

18.02 

21.48 

9.28 

20.15 

OH-58F  Simulation  #2 

8.86 

21.32 

18.91 

19.23 

SH-3H 

14.70 

20.00 

12.40 

18.80 

OH-58F  Simulation  #1 

16.43 

12.21 

10.05 

15.16 

RAH-66  EDS 

11.84 

14.98 

4.54 

13.25 

CH-53F 

7.50 

10.50 

7.40 

10.00 

RAH-66  CPC 

3.29 

12.94 

7.89 

9.80 

UH-60M  (LEUE) 

6.36 

11.81 

3.09 

9.15 

AH-64D  -  IUAS  (RACRS) 

9.01 

7.58 

4.64 

8.51 

UH-60M  (EUD) 

13.88 

6.89 

0 

8.50 

OH-58F  Simulation  #3 

7.95 

7.16 

3.09 

7.48 

CH-46E 

5.40 

7.80 

4.50 

7.00 

AH-64D  VUIT-2  (RACRS) 

3.18 

5.05 

4.64 

4.98 

3.6  Crewstation  Design  Enhancements 

Several  design  enhancements  to  the  OH-58F  crewstation  displays  have  been  made  via  the 
Crewstation  Working  Group  and  crewstation  simulation  assessment  process  (see  figure  7).  These 
enhancements  improved  the  functionality  and  presentation  of  display  pages  to  pilots  and  overall 
crewstation  interface.  Examples  of  the  enhancements  are  improved  color-coding  of  battlefield 
graphics,  reduced  number  of  button  presses  to  display  information  on  MFDs,  refinement  of  the 
composite  map  page,  and  enhanced  presentation  of  operational  limits  on  the  Systems  Page. 
Additionally,  the  Crewstation  Working  Group  and  simulation  process  has  aided  in  the  refinement 
of  TTP  for  OH-58F  operational  employment. 
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Added  capability  to  view  additional 
information  on  map  display 
(e  g.,  operational  limits)  to  increase 
pilot  situational  awareness 


Changed  bezel  button  logic  for 
map  and  sensor  display  modes 
to  enhance  usability 


Figure  7.  Examples  of  crewstation  design  enhancements. 


4.  Summary 


4.1  Crew  Workload 

The  pilots  and  copilots  reported  that  workload  was  manageable  for  all  flight  and  mission  tasks 
they  performed  during  the  missions  to  include  LOI  2  tasks  with  the  UAS.  The  SMEs  reported 
that  workload  was  manageable  for  the  pilots  and  copilots.  The  average  mission  workload  ratings 
provided  by  the  pilots,  copilots,  and  SMEs  were  lower  than  the  Objective  (5.0)  and  Threshold 
(6.0)  BWRS  workload  rating  requirements  contained  in  the  OH-58F  CDD. 

4.2  Visual  Workload 

The  copilots  typically  spent  4%  of  the  time  visually  focused  OTW  during  VFR  flight.  The  small 
percentage  of  time  that  the  copilots  were  visually  focused  outside  the  aircraft  was  mostly  due  to 
the  workload  required  to  manage  information  on  the  crewstation  displays,  operate  the  NMS  and 
the  lack  of  in-depth  experience  that  the  copilots  had  with  the  crewstation  interface.  The  pilots 
typically  spent  77%  of  the  time  visually  focused  OTW  during  VFR  flight.  The  amount  of  time 
(23%)  that  the  pilots  were  visually  focused  inside  the  aircraft  was  due  to  instrument  scans,  lack 
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of  in-depth  experience  with  the  crewstation  interface,  and  the  “helping  behaviors”  of  the  pilots 
when  flying  the  aircraft.  The  pilot  occasionally  helped  the  copilot  manage  infonnation  on  the 
crewstation  displays  that  kept  both  of  them  visually  focused  inside  the  crewstation. 

4.3  Crew  SA 

The  pilots  and  copilots  reported  that  they  typically  experienced  good  SA  during  missions,  were 
able  to  maintain  knowledge  of  the  aircraft  energy  state,  tactical  environment,  and  mission,  were 
able  to  partially  anticipate  and  accommodate  trends,  and  had  minimal  task  shedding  (due  to  high 
workload)  during  the  missions.  They  reported  that  they  had  fairly  high  to  very  high  levels  of  SA 
of  most  battlefield  elements  (e.g.,  location  of  their  ownship,  route  information)  during  the 
missions.  The  SMEs  reported  that  the  crews  typically  had  adequate  levels  of  SA  during  missions. 

4.4  Pilot-Crewstation  Interface  (PCI) 

The  pilots  reported  they  were  able  to  effectively  use  the  MFD  pages  and  functions,  quickly 
navigate  through  the  pages,  subpages  and  overlays  on  the  crewstation  displays,  easily  use  the 
switches  on  the  cyclic  and  collective,  and  easily  use  the  switches  to  control  the  NMS.  They  also 
reported  that  it  was  easy  to  detect  the  Warnings,  Cautions,  and  Advisories  on  the  MFD  and  entry 
into  operational  limits.  The  pilots  reported  that  the  CDS5  software  was  quicker  and  easier  to  use 
versus  the  CDS4  software.  They  commented  that  the  6><8  display  enhanced  SA  and  the  NMS 
functionality  is  an  improvement  versus  the  MMS  on  the  OH-58D  Kiowa  Warrior. 

4.5  Mission  Success 

All  of  the  missions  performed  by  the  aircrews  were  rated  as  “successful”  by  the  SMEs  who 
observed  each  mission. 

4.6  Simulator  Sickness 

The  pilots  and  copilots  reported  that  they  typically  experienced  very  mild  to  mild  simulator 
sickness  symptoms  during  the  evaluation.  The  OH-58F  simulator  did  not  induce  debilitating 
simulator  sickness  symptoms  and  should  continue  to  be  a  suitable  simulation  environment  for 
future  assessments  and  training. 

4.7  Simulator  Functionality 

Simulator  functionality  was  somewhat  limited  during  the  HFE  #3  assessment.  The  infonnation 
and  data  listed  in  the  Results  and  Summary  sections  of  this  report  should  be  interpreted  based  on 
these  limitations. 

4.8  HFE  #1,  HFE  #2,  and  HFE  #3  Crewstation  Assessments 

The  aircrew  workload,  SA,  and  crewstation  interface  data  collected  during  the  HFE  #1,  HFE  #2, 
and  HFE  #3  assessments  have  been  helpful  in  evaluating  and  refining  the  crewstation  design  and 
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refining  TTP.  These  assessments  have  helped  ensure  that  the  aircraft  will  meet  requirements  as  it 
progresses  toward  operational  testing  and  fielding. 


5.  Recommendations 


The  following  recommendations  are  made  to  enhance  the  overall  effectiveness  and  suitability  of 
the  OH-58F  crewstation  development  and  assessment  process: 

•  Workload,  SA,  and  crewstation  interface  data  should  be  collected  in  the  same  format  (as 
the  crewstation  assessments)  during  operational  testing.  This  will  allow  direct  comparison 
to  assess  workload,  SA,  and  the  crewstation  interface. 

•  Continue  to  resolve  the  PCI  issues  identified  during  the  HFE  simulation  assessments. 

•  Consider  using  the  OH-58F  simulator  to  augment  pilot  training  for  the  Limited  User  Test 
(LUT)  and  Initial  Operational  Test  and  Evaluation. 
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Appendix  A.  Bedford  Workload  Rating  Scale  Scores  and  Pilot  Comments 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


BEDFORD  WORKLOAD  RATING  SCALE  (BWRS) 


Was  workload 
satisfactory 
without  reduction  in 
spare  (workload)  capacity? 


NO 


Was  workload  tolerable 
for  the  task? 


NO 


Was  it  possible  to 
complete  the  task? 


Pilot  Decisions 


Workload  Description 


“Rating’ 


Workload  insignificant 

1 

Workload  low 

2 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Task  abandoned.  Pilot  unable  to 
apply  sufficient  effort 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 

4 

Reduced  spare  capacity.  Additional 

tasks  cannot  be  given  the  desired 

5 

amount  of  attention 

Little  spare  capacity:  level  of  effort 

allows  little  attention  to  additional 

6 

tasks 

Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 

7 

Very  high  workload  with  almost  no 
spare  capacity.  Difficulty  in 
maintaining  level  of  effort 

8 

Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 

9 

10 
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BEDFORD  WORKLOAD  RATING  SCORES 


Task 

No. 

Flight  and  Mission  Tasks 

1026 

Maintain  Airspace  Surveillance 

1028 

Perfonn  Hover  Power  Check 

1030 

Perform  Hover  Out-Of-Ground-Effect  (OGE)  Check 

1032 

Perfonn  Radio  Communication  Procedures 

1038 

Perform  Hovering  Flight 

1040 

Perform  Visual  Meteorological  Conditions  (VMC) 
Takeoff 

1044 

Navigate  by  Pilotage  and  Dead  Reckoning 

1046 

Perform  Electronically  Aided  Navigation 

1048 

Perfonn  Fuel  Management  Procedures 

1052 

Perform  VMC  Flight  Maneuvers 

1058 

Perform  VMC  Approach 

1066 

Perform  A  Running  Landing 

1070 

Respond  to  Emergencies 

1074 

Respond  to  Engine  Failure  in  Cruise  Flight 

1140 

Perform  Nose  Mounted  Sensor  (NMS)  Operations 

1142 

Perfonn  Digital  Communications 

1155 

Negotiate  Wire  Obstacles 

1170 

Perfonn  Instrument  Takeoff 

1176 

Perfonn  Non  Precision  Approach  (GCA) 

1178 

Perfonn  Precision  Approach  (GCA) 

1180 

Perfonn  Emergency  GPS  Recovery  Procedure 

1082 

Perfonn  an  Autorotation 

Pilot  Workload 
Rating 


Copilot 

Workload 

Rating 


Perform  Unusual  Attitude  Recovery 


Operate  ASE/transponder 


Respond  to  IMC  Conditions 


Perfonn  Refueling  /  Rearming  Operations 


Perform  Electronic  Countenneasures  /  Electronic 
Counter-Countermeasures 


Transmit  Tactical  Reports 


Perfonn  Terrain  Flight  Takeoff 


Perfonn  Terrain  Flight 


Perform  Terrain  Flight  Approach 


Perform  Masking  and  Unmasking 


Perfonn  Terrain  Flight  Deceleration 


Perform  Actions  on  Contact 


Perform  Weapons  Initialization  Procedures 


Perfonn  Firing  Techniques 


Engage  Target  with  .50  Cal 


Engage  Target  with  Hellfire 


Engage  Target  with  Rockets 


Perform  Aerial  Observation 


Perform  Target  Handover 


Call  for  Indirect  Fire 


Perform  Multi-Aircraft  Operations 


Perfonn  Combat  Maneuvering  Flight 


Perform  Close  Combat  Attack 


Perform  Combat  Position  Operations 


Call  for  Tactical  Air  Strike 


Zone  Reconnaissance 


Route  Reconnaissance 


Area  Reconnaissance 


Level  2  UAS 
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Aerial  Surveillance 

2.56 

2.50 

Overall  Workload  for  the  Mission 

2.56 

3.44 

Pilot  Workload  Comments: 

Comments  for  flight  and  mission  tasks  that  were  given  a  workload  rating  of  ‘5’  or  higher  during 
missions: 

•  Tasks  1070  (Respond  to  Emergencies),  1074  (Respond  to  Engine  Failure  in  Cruise 
Flight),  and  1182  (Perform  Unusual  Attitude  Recovery)  are  workload  intensive  allowing 
no  additional  tasks  to  be  performed  to  insure  safe  outcome  of  maneuver. 

•  Tasks  1070  (Respond  to  Emergencies),  1074  (Respond  to  Engine  Failure  in  Cruise 
Flight),  and  1182  (Perform  Unusual  Attitude  Recovery)  were  experienced  in  responding 
to  engine  failure  in  cruise  flight.  In  order  to  establish  safe  outcome  of  maneuver  complete 
attention  was  focused  on  rotor  rpm,  airspeed  and  sequencing  of  final  flight  maneuvers. 

No  additional  tasks  are  recommended  while  accomplishing  this  task  without  inviting 
jeopardy  to  maneuver  outcome.  Only  recommendation  to  reducing  workload  in  cruise 
flight  autorotation  would  be  to  add  symbology  around  airspeed  numbers  only  when  in  an 
autorotation  when  airspeed  falls  outside  of  safe  limits  (55-80  kts). 

•  Task  1074  (Respond  to  Engine  Failure  in  Cruise  Flight)  as  soon  as  the  warnings  started,  I 
was  focused  only  on  successfully  completing  the  maneuver.  I  shed  all  radio  calls  to  my 
left  seater  and  stayed  focused  on  my  landing  area,  instruments  and  controls. 

•  Tasks  1 170  (Performing  Instrument  Take  Off)  and  1184  (Respond  to  IMC  Conditions) 
were  encountered  simultaneously  during  take-off  from  a  FARP  which  resulted  in 
unexpected  brownout  condition.  Safe  outcome  of  flight  without  overtorque  to  aircraft 
required  one  pilot  to  commit  to  VSD  display  while  other  pilot  not  on  the  controls 
alternated  attention  to  SYS  display  page  (torque  setting  primarily),  outside  horizon  (to 
scan  for  VMC  conditions),  and  the  occasional  backup  of  pilot  VSD  screen.  No  additional 
tasks  are  safely  pennitted  under  such  an  event  to  ensure  successful  outcome. 

•  Tasks  1 170  (Performing  Instrument  Take  Off),  1184  (Respond  to  IMC  Conditions)  and 
1407  (Perfonn  Terrain  Flight  Takeoff)  were  experienced  in  Brown-out  takeoff  condition. 
Similar  to  above,  power  setting,  sequencing  of  flight  maneuvers,  and  aircraft  attitude 
become  primary  focus  with  little  ability  to  focus  on  additional  tasks.  Difficulty  of  task  is 
not  affected  by  software  to  OH-58. 

•  Task  1413  (Perfonn  Actions  on  Contact)  when  hostile  threat  is  encountered  requires 
significant  focus  to  evade  enemy  fire  and  develop  situation  in  a  manner  to  neutralize 
threat  or  bypass.  New  aircraft  systems  reduce  workload  on  aircrew  compared  to  current 
OH-58D  CDS4  Block  2  software  and  systems. 

•  Tasks  1413  (Perform  Actions  on  Contact)  when  I  began  taking  fire  from  such  a  close 
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range,  my  only  focus  was  suppressing  and  getting  distance  between  the  aircraft  and  the 
target.  I  was  able  to  make  a  couple  of  radio  calls  to  the  trail  aircraft  until  we  were  safely 
away  and  set  up  to  engage  the  target. 

•  Task  1471  (Perfonn  Target  Handover)  using  UAS  reduces  additional  workload  capacity 
as  2  of  the  3  screens  within  cockpit  as  well  as  single  radio  become  utilized  to  designate 
target  to  be  handed  over,  confirmation  that  correct  target  was  handed  over,  and  verbal 
confirmation  of  spot  and  track  of  target.  Redundancy  of  confirmation  however  insures 
PID  and  minimizes  potential  for  collateral  damage  and  fratricide/non-combatant 
casualties. 

Miscellaneous  Workload  Comments: 

•  The  left  seater  in  this  profile  is  focused  inside  90%  of  the  time  operating  the  systems. 
Maintain  airspace  surveillance  is  not  being  accomplished.  Lack  of  experience  with  the 
NMS  and  UAS  video  also  contributed  to  higher  workload. 

•  Overall  workload  was  a  ’5’  mostly  due  to  inexperience. 

•  While  trying  to  maneuver  the  aircraft  and  access  the  UAS  page  as  well  as  maneuver  the 
UAS  drastically  increased  the  workload  on  the  right  seater. 

•  Due  to  lack  of  experience  with  the  NMS  workload  was  high.  Lack  of  situational 
awareness  was  experienced  because  I  became  so  focused  on  operating  the  NMS. 
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Appendix  B.  Subject  Matter  Expert  Workload  Comments 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


27 


•  Mission  1  -  CPG  had  little  capacity  to  perfonn  standard  crew  tasks  due  addition  of  new 
functions.  CPG  is  a  junior  flight  crew  member  who  would  have  had  difficulty 
perfonning  nonnal  operations  in  a  CDS4  environment.  Right  seat  was  more  than 
capable  of  making  up  for  the  deficiencies  of  the  left  seat. 

•  Mission  1  -  Left  seat  crewmember's  experience  level  with  CDS4  contributed  to  his 
difficulties  in  utilizing  CDS5  effectively.  Left-seater  was  not  familiar  enough  with 
CDS5  capabilities  in  order  for  him  to  use  it  to  his  advantage  during  setup  as  well  as 
during  the  mission.  Unable  to  use  particular  facets  of  the  system  such  as  Laser  Spot 
Tracker,  EOS,  or  MUM  without  being  prompted  by  right-seater.  Right  seat 
crewmember's  workload  was  only  slightly  increased  due  to  left  seater's  difficulty  in 
completing  tasks,  but  did  not  affect  his  ability  to  conduct  operations. 

•  Mission  1  -  Left  Seat  Crewmember  was  behind  the  aircraft  and  had  significant  issues 
multitasking  as  scenario  workload  increased.  Increased  training  will  help  this  junior 
aviator  increase  proficiency  in  working  left  seat  mission  equipment  and  radios.  Right 
seat  PC  directed  left  seater  as  workload  increased  and  basic  tasks  needed  additional 
attention. 

•  Mission  3  -  Crew  had  sufficient  capacity  but  at  times,  left-seater  had  to  drop  certain 
tasks  such  as  response  to  BFT  messages  and  making  radio  calls. 

•  Mission  3  -  Left  Seat  Crewmember's  task  saturation  increased  the  workload  for  the 
Right  Seater  attempting  to  maintain  or  increase  SA.  Left  Seat  Crewmember  sometimes 
directed  trail  aircraft  to  take  simple  tasks  like  BFT  messages  and  shed  those  tasks 
altogether,  and  not  returning  to  them  even  when  lower  workloads  were  present.  Both 
crewmembers  lost  SA  at  times,  however  it  didn’t  significantly  affect  the  mission. 

•  Mission  6  -  CPG  had  NMS  control  issues  (not  associated  with  the  SIM)  that  caused  a 
snowball  effect  for  him  in  the  cockpit.  Result  being  the  lack  of  situational  awareness 
coupled  with  being  two  steps  behind  for  the  duration  of  the  mission. 

•  Mission  7  -  Right  seat  workload  was  increased  due  to  left  seat  limited  experience  level. 
This  caused  right  seat  to  both  fly  and  monitor  left  seat  workload  to  ensure  tasks  were 
accomplished.  Right  seat  was  still  able  to  use  system  tools  (i.e.  6x8  map)  as  needed. 

•  Mission  7-1  think  the  overall  aviation  experience  of  the  left  seater  caused  the  overload 
situation.  Right  seater's  workload  was  increased  with  trying  to  direct  the  left  seater  in 
specific  tasks 

•  Mission  8  -  This  was  a  much  more  experienced  crew.  Use  of  the  new  systems  did  not 
affect  workload. 

•  Mission  9  -  Additional  familiarity  with  new  systems  is  helping  to  reduce  initial 
workload  observations.  The  addition  of  the  6x8  screen  and  MUM  capability  will 
undoubtedly  increase  left-seat  workload,  but  this  crew  handled  it  well. 
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Appendix  C.  Situation  Awareness  Ratings  and  Comments 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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China  Lake  Situational  Awareness  Rating  Scale  (CLSA) 


pan 


Copilot  M 
Rating 
2.22 
(SD  =  109) 


VERY  GOOD 
1 


GOOD 

2 


ADEQUATE 

3 


POOR 

4 


VERY  POOR 
5 


Full  knowledge  of  aircraft  energy  state,  tactical  environment  and 
mission 

Full  ability  to  anticipate  and  accommodate  trends 


Full  knowledge  of  aircraft  energy  state,  tactical  environment  and 
mission 

Partial  ability  to  anticipate  and  accommodate  trends 
No  task  shedding 


Full  knowledge  of  aircraft  energy  state,  tactical  environment  and 
mission 

Saturated  ability  to  anticipate  and  accommodate  trends 
Some  shedding  of  minor  tasks 


Fair  knowledge  of  aircraft  energy  state,  tactical  environment  and 
mission 

Saturated  ability  to  anticipate  and  accommodate  trends 
Shedding  of  all  minor  tasks  as  well  as  many  not  essential  to  flight 
safety  and  mission  effectiveness 


Minimal  knowledge  of  aircraft  energy  state,  tactical  environment  and 
mission 

Oversaturated  ability  to  anticipate  and  accommodate  trends 
Shedding  of  all  tasks  not  absolutely  essential  to  flight  safety  and 
mission  effectiveness 


lot  Mean 
Rating 
1.67 

D  =  0.71) 
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Battlefield 

Elements 

Very  High 
Level  of 
Situation 
Awareness 

Fairly  High 
Level  of 
Situation 
Awareness 

Borderline 

Fairly  Low 
Level  of 
Situation 
Awareness 

Very  Low 
Level  of 
Situation 
Awareness 

Right 

Seat 

Left 

Seat 

Right 

Seat 

Left 

Seat 

Right 

Seat 

Left 

Seat 

Right 

Seat 

Left 

Seat 

Right 

Seat 

Left 

Seat 

Location  of 
Enemy  Units 

44  % 

44% 

44% 

44% 

11  % 

11  % 

0% 

0% 

0% 

0% 

Location  of 
Friendly  Units 

66% 

45  % 

33  % 

55% 

0% 

0% 

0% 

0% 

0% 

0% 

Location  of  My 
Aircraft  During 
Missions 

66% 

45  % 

33  % 

55% 

0% 

0% 

0% 

0% 

0% 

0% 

Location  of 
Cultural 
Features  (e.g., 
bridges) 

33  % 

25% 

66% 

75% 

0% 

0% 

0% 

0% 

0% 

0% 

Route 

Information 
(ACPs,  BPs, 
EAs,  RPs,  etc.) 

33  % 

11  % 

66% 

88% 

0% 

0% 

0% 

0% 

0% 

0% 

Status  of  My 
Aircraft 
Systems  (e.g., 
fuel 

44  % 

22% 

55% 

67% 

0% 

11  % 

0% 

0% 

0% 

0% 

Location  of 
Other  Aircraft 
During  Missions 

38  % 

33  % 

50% 

44% 

13  % 

22% 

0% 

0% 

0% 

0% 

Pilot  Situational  Awareness  Comments 

•  Even  though  we  had  radio  contact  with  the  BlackHawk  flight,  I  didn’t  notice  their  BFT 
icons.  I  only  had  them  in  sight  when  they  were  up  and  out  of  the  LZ.  Also,  because  we 
coordinated  the  A- 10  flight  through  trail,  my  awareness  of  their  position  and  activity  was 
low. 

•  Multiple  times  I  lost  enemy  personnel  while  working  the  NMS. 

•  For  other  aircraft,  I  had  a  general  idea  where  the  UAV  was  operating  but  I  never  utilized 
the  feature  on  the  UAS  page  to  find  out  his  exact  position  because  I  was  usually  busy 
working  the  NMS.  For  status  of  aircraft  systems,  I  placed  a  higher  priority  on  executing 
the  mission  with  the  NMS,  UAV,  and  radios  and  mentally  placed  the  aircraft  status  in  the 
backseat,  probably  due  to  the  exercise  being  a  simulation.  I  also  am  not  as  comfortable 
yet  only  having  1  driving  PWR  gauge. 
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•  In  the  second  of  the  three  engagements  of  this  flight,  I  did  not  understand  my  aircraft 
position  relative  to  the  3  fighters  and  vehicle  until  we  were  being  fired  upon  and  my  left 
seater  pointed  at  them  at  12  o’clock  through  the  windscreen.  I  may  have  confused  UAV 
feed  with  the  NMS  or  vice  versa,  and  up  to  that  point  my  left  seater  was  the  one  tracking 
those  targets,  I  should  have  prompted  him  to  help  me  get  oriented  much  sooner. 

SME  Situational  Awareness  Comments 

•  UAS  asset  was  constantly  behind  the  aircraft  causing  loss  of  signal  feed  on  a  constant 
basis.  NMS  usage  was  not  adequate  for  the  mission  and  the  CPG  failed  to  properly 
perform  basic  functions  that  would  have  enabled  better  mission  success.  SA  was  lost  by 
both  the  PC  and  PI  on  two  occasions  at  key  times  during  the  mission.  Prompting  by  the 
UAS  operator  was  critical  to  getting  the  aircraft  re-focused  on  mission  objectives. 

•  Right-seater  picked  up  several  left-seat  tasks  in  order  to  assist  left-seater,  which 
reduced  pilot  SA  on  multiple  occasions  -  one  resulted  in  the  aircraft  losing  its  primary 
target,  another  resulted  in  the  aircraft  overflying  an  NAI,  causing  it  to  be  fired  upon  by 
enemy  forces.  Left  seater  was  almost  completely  unaware  of  ownship  location  due  to 
difficulties  with  operating  the  NMS  and  focusing  on  UAV  targets.  Excessive  use  of  the 
UAS  screen  (when  no  feed  was  available)  reduced  pilot  use  of  CDS5  maps,  also 
reducing  SA.  Good  use  of  the  elevation  banding  was  critical  in  preventing  Controlled 
Flight  into  Terrain  (CFIT)  when  the  crew  went  IIMC. 

•  In  the  middle  of  the  mission,  the  crew  was  fixated  utilizing  the  sensor  in  a  narrow  FOV 
and  lost  SA  with  surrounding  and  flew  directly  into  an  ambush  alongside  a  road  that 
could  have  been  detected.  Crew  split  their  flight,  sending  UAS  to  SP  of  RTE  Coral  and 
SWT  would  conduct  Air  Route  Recon  from  RP  to  SP.  Later  in  the  mission  crew  was 
fixated  on  the  acquiring  a  ZSU  23-4  that  had  pinged  their  ASE.  Due  to  Actions  on 
Contact,  crew  lost  SA  on  ZSU  23-4's  location.  On  attempting  to  reacquire,  SWT 
paralleled  the  moving  enemy  element  and  eventually  turned  into  the  path  and  moved  to 
within  1km  of  the  enemy  location.  SWT  would  have  been  KIA  if  engaged  by  the 
threat. 

•  Left-seat  pilot  had  good  SA  of  the  battlefield.  Right-seater  relied  more  on  left-seater 
than  on  CDS  for  SA  -  did  not  fully  utilize  the  system  for  navigation  or  SA  of  other 
entities  on  the  battlefield.  Right-seater  came  inside  the  cockpit  more  than  once  to 
accomplish  left-seat  tasks  (but  not  due  to  task  saturation).  In  doing  so,  crew  temporarily 
lost  positive  ID  of  enemy  (due  to  hostile  fire  or  evasive  action)  on  multiple  occasions 
and  had  some  difficulty  re-acquiring  with  the  NMS.  Good  use  of  the  MUM  system  for 
acquiring  targets  and  also  transmitting  own-ship  feed  to  the  ground  forces. 

•  Crew  had  SA,  but  task  saturation  in  more  than  one  instance  caused  left-seater  to 
abandon  tasks.  Left-seater's  difficulty  in  communication  with  role  players  caused  right- 
seater  to  take  over  tasks  more  than  once.  Right-seater  was  consistently  aware  of  all 
entities  on  the  battlefield. 
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•  Due  to  Left  Seat  proficiency,  Right  Seater  was  brought  into  the  cockpit  to  direct  tasks 
and  lost  some  SA,  especially  when  brought  into  the  cockpit  to  help  direct  tasks  needed 
to  be  accomplished  by  the  Left  Seater 

•  Crews  effectively  used  standoff  with  the  NMS  and  CDS5  system  tools 

•  Pilot  was  focused  on  being  inside  watching  the  CPG  much  more  than  would  normally 
be  expected.  Loss  of  SA  was  noted  at  several  times  due  to  this  fact,  as  well  as  not 
having  ground  reference  points  to  keep  track  of  the  aircraft  location. 

•  Right  seat  increased  workload  resulted  in  reduced  SA.  At  one  point  where  right  seat 
was  unable  to  identify  current  position  which  could  have  resulted  in  a  fratricide 
incident  -  right  seat  did  a  good  job  compensating  by  using  system  tools  for  SA,  but  it 
brought  him  in  the  cockpit  for  longer  periods  than  necessary.  Left  seat  limited 
experience  level  also  reduced  SA  of  friendly  and  enemy  entities.  One  significant  SA 
event  occurred  when  Close  Air  Support  was  called  in  to  drop  ordnance  on  an  enemy 
location.  Pilots  did  not  consider  UAS  location  with  reference  to  the  CAS  inbound  run 
or  gun-target  line.  This  mission  had  many  more  friendly  and  enemy  entities  in  contact 
which  required  much  more  SA  in  order  to  be  effective  and  avoid  getting  shot  down. 

The  crew  was  aware  of  this  for  the  most  part  and  attempted  to  use  additional  assets  to 
maintain  SA,  but  at  times,  the  crew  was  close  to  sensory  overload  -  this  was  more  due 
to  a  single  aircraft  performing  the  mission  of  two  during  simulation  -  it  may  not  be 
realized  to  this  extent  in  actual  flight  except  for  extreme  situations. 

•  Both  crewmembers  were  consistently  behind  the  aircraft  and  lost  SA  on  numerous 
occasions.  Left  seat  failed  to  utilize  NMS  to  full  potential  and  often  because  of  lack  of 
SA  were  scanning  behind  the  convoy.  Left  seater  also  shed  tasks  (BFT  messages)  and 
never  returned  to  them  as  workload  decreased.  Important  mission  information.. such  as 
a  weather  warning.  Right  seater  took  control  of  the  6x8  for  map  display  and  never 
utilized  his  5x7  HSD  for  SA.  Left  seater  was  task  saturated  and  became  overloaded  due 
to  trying  to  maintain  control  of  the  UAS. 

•  Crew  had  reduced  SA  based  on  increased  workload  and  failure  to  utilize  systems  for 
accurate  SA. 

•  SA  of  friendly  and  enemy  was  no  issue,  however  when  calling  CAS  and  artillery,  crew 
did  not  take  UAV  location  into  account.  SA  during  IIMC  was  improved  with  use  of 
terrain  elevation  banding. 

•  The  crew  was  denied  use  of  their  UAS  initially,  which  did  not  have  any  significant  SA 
effect  until  on  the  objective.  Once  on  the  objective,  crew  struggled  with  SA  of  both 
friendly  and  enemy  forces.  An  additional  UAS  was  available  as  an  ISR  asset,  which 
would  have  greatly  increased  SA  if  it  were  utilized.  One  specific  SA  factor  was 
direction  of  artillery  fires  and  inbound  heading  of  CAS.  Crew  did  not  have  the  UAS 
directly  tasked  to  them,  so  SA  of  its  location  was  not  considered  during  fire  missions. 
CDS5  has  multiple  tools  available  to  aid  in  preventing  this  situation. 
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•  Based  on  assigned  mission,  crew  was  completely  focused  on  LZ  and  didn’t  utilize 
available  UAS  assets  to  increase  SA  on  the  objective  which  would  have  increased 
battlefield  SA  to  threats  within  the  objective  area.  Crew  engaged  two  personnel  around 
the  LZ  with  a  Hellfire  Missile.  Once  SA  of  threat  increased,  crew  occasionally  used 
UAS  assets  to  increase  SA. 

•  Pilot  was  looking  at  the  6x8  more  than  he  should  be  which  causes  both  pilots  to  be 
inside  and  subsequent  loss  of  SA.  Pilot  tried  to  perfonn  unmasking  procedures  with  the 
NMS  which  actually  causes  the  aircraft  to  completely  unmask. 

•  Minor  SA  issues  with  this  crew.  Left-seater  maximized  use  of  CDS5  new  systems.  At 
times,  the  right-seater  would  target  fixate  on  the  NMS  or  UAV  screens,  reducing  his 
SA  of  the  aircraft  position.  In  an  actual  flight,  proprioceptive  senses  would  alert  the 
pilot  to  these  instances. 

•  Crew  coordination  was  a  slight  issue  when  dealing  with  announcing  actions.  Pilot 
became  disoriented  due  to  lack  of  crosstalk  with  CPG. 

•  Crews  unfamiliar  with  MUM  are  not  utilizing  it  as  an  effective  SA  tool.  Crew  came 
close  to  international  border  during  mission.  This  could  have  been  avoided  if  SA  tools 
were  used  such  as  aircraft  map  or  MUM  map.  Having  a  third  screen  is  also  new  to  the 
crew,  so  maximizing  use  of  all  three  screens  may  require  additional  training  until  a 
standard  procedure  is  developed. 

•  MUM  Teaming  experience  and  cockpit  MUM  proficiency,  coupled  with  MUM  being  a 
federated  system  which  causes  the  crew  to  be  heads  down  inside  the  cockpit  to  transfer 
grids  to  the  CDS  from  the  MUM  system,  significantly  reduces  overall  effectiveness  of 
the  CDS  MUM  system  as  compared  to  an  integrated  system  which  would  have  all  of 
the  BFT/MUM/Targeting  information  all  in  one  screen. 

•  Crew  focused  on  following  a  vehicle  and  lost  SA  to  the  surrounding  areas  that  affected 
their  safety.  Loss  of  visual  cues  and  reference  to  the  ZSU's  after  initial  contact  led  them 
to  be  engaged  within  1  km. 
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Appendix  D.  Pilot-Crewstation  Interface  (PCI)  Ratings  and  Comments 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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PV1.  The  following  table  lists  the  components  (e.g.,  display  pages,  sub-pages,  overlays)  of 
the  KW  CASUP  crewstation.  For  each  component,  indicate  whether  or  not  you  experienced 
a  problem  using  the  component  in  a  quick  and  efficient  manner  during  the  missions.  Check 
‘Yes’  if  you  experienced  one  or  more  problems.  Check  ‘No’  if  you  did  not  experience  any 
problems.  Check  ‘Not  Used’  if  you  did  not  use  the  component  during  the  simulation. 

•  Multifunction  Displays  (MFD) 


o  Vertical  Situation  Display 

Yes  0% 

No  100% 

o  Horizontal  Situation  Display 

Yes  0% 

No  100% 

o  NMS  Operations 

Yes  17% 

No  83% 

o  COM  Pages 

Yes  17% 

No  83% 

o  NAV  Pages 

Yes  0% 

No  100% 

o  Digital  Map 

Yes  17% 

No  83% 

o  Engine  Instruments 

Yes  0% 

No  100% 

o  ASE  Pages 

Yes  0% 

No  83% 

Not  Used  17% 

o  Weapons  Pages 

Yes  0% 

No  0% 

Not  Used  100% 

o  Managing  GPS/Flight  Plan 

Yes  0% 

No  100% 

o  Mission  Page 

Yes  0% 

No  100% 

o  Warning,  Caution,  Advisory  Disp.  Yes  0% 

No  100% 

o  ‘Direct  To’ 

Yes  0% 

No  100% 

o  ACP  Function 

Yes  0% 

No  100% 

o  Fuel  Management  Pages 

Yes  0% 

No  100% 

o  Data  Function 

Yes  0% 

No  0% 

Not  Used  100% 

o  ACP  Function 

Yes  0% 

No  100% 

o  UAS  Function 

Yes  0% 

No  100% 

o  Autopaging 

Yes  0% 

No  100% 
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If  you  answered  “Yes”  to  any  of  the  questions,  describe  a)  the  problems  you  experienced, 
b)  how  much  the  problems  degraded  your  performance,  and  c)  any  recommendations  you 
have  for  improving  the  design  of  the  components. 

Pilot  comments: 

•  Com  pages  were  difficult  for  me  to  get  a  feel  for  at  first  but  now  I  feel  more  comfortable 
with  it. 

•  Lack  of  experience  with  the  NMS  caused  errors  in  operation,  b)  Caused  task  saturation 
occasionally)  It’s  not  a  design  problem  it’s  an  experience  issue. 

•  Digital  Map  on  6x8  screen  does  not  display  compass  rose  similar  to  HSD  pages  on  5x7 
screens.  Incorporating  this  feature  to  the  6x8  HSD  will  significantly  assist  crew  by 
displaying  relevant  heading  information  and  contributing  to  greater  situational 
awareness.  Recommendation  for  VSD  screen  in  the  event  of  engine  out  or  autorotation  at 
cruise  scenario  would  be  to  include  noticeable  symbology  (boxed  numbers  or  ascetics)  by 
airspeed  number  if  out  of  tolerance  for  safe  outcome  of  emergency  procedure. 


PV2.  How  quickly  were  you  able  to  navigate  through  the  pages,  sub-pages  and/or  overlays 
for: 


Vertical  Situation  Display  (Circle  one) 


Avg.  Rating 
1.33 

2 

3 

4 

5 

Very  * 

Somewhat 

Borderline 

Somewhat 

Very 

Quickly 

Quickly 

Slowly 

Slowly 

Horizontal  Situation  Display  (Circle  one) 

1 

Avg.  Rating 
2.17 

3 

4 

5 

Very 

Some^/hat 

Borderline 

Somewhat 

Very 

Quickly 

Quickly 

Slowly 

Slowly 

Digital  Map  (Circle  one) 

1 

Avg.  Rating 
2.17 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Quickly 

Quickly 

Slowly 

Slowly 

37 


Engine  Instruments  (Circle  one) 


Avg.  Rating  2 

3 

4 

5 

1.67 

Very  A  Somewhat  Borderline  Somewhat  Very 

Quickly  Quickly  Slowly  Slowly 


If  you  answered  ‘Somewhat  Slowly’,  or  ‘Very  Slowly’  to  any  of  the  questions,  list  the 
component  and  why  navigation  was  slow  (e.g.,  ‘navigating  the  menu  system  on  the  digital 
map  was  a  slow  process  due  to  having  to  page  through  several  screen  displays’). 

Pilot  comments: 

•  Working  the  HSD  and  Map  in  conjunction  with  the  NMS  page  and  Sight  Mode  selection 
knob  really  does  bog  down  the  system.  I  thought  this  was  a  simism  and  hope  it  doesn’t 
transfer  to  the  aircraft  (simulator  issue). 

PV3.  Please  answer  the  following  questions  regarding  the  Nose  Mounted  Sensor  (NMS). 
PV3-1.  Did  you  experience  any  problems  using  the  following  NMS  switches/controls? 


0  Sensor  Select 

Yes  17% 

No  83% 

0  FOV  Select 

Yes  17% 

No  83% 

0  Laser 

Yes  0% 

No  100% 

0  LOS  Designate 

Yes  0% 

No  0% 

NA  100% 

0  Range/Polarity 

Yes  0% 

No  83% 

NA  17% 

0  Manual  Slave 

Yes  17% 

No  83% 

0  Point  Track 

Yes  0% 

No  100% 

If  you  answered  “Yes”  to  any  of  the  questions,  describe  a)  the  problems  you  experienced, 
b)  how  much  the  problems  degraded  your  performance,  and  c)  any  recommendations 
you  have  for  improving  the  design  of  the  NMS  switches/controls. 

Pilot  comments: 

•  Pressing  the  Man/Slave  button  quickly  while  using  the  LOS,  NMS  page  and  mode  select 
switch  bogs  down  the  system  pretty  regularly  (simulator  issue). 

•  Occasionally  I  would  use  the  wrong  switch)  caused  slight  delay  in  achieving  the  desired 
result)  practice. 
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PV4.  Did  you  have  difficulty  using  any  of  the  switches  on  the  collective  or  cyclic  grips? 


Collective  Grip 

Yes  0% 

No  100% 

Cyclic  Grip 

Yes  0% 

No  100% 

If  you  answered  “Yes”  for  either  flight  control,  please  list  which  flight  control  and  switch(es), 
and  the  problems  you  experienced  (e.g.,  confused  two  switches  due  to  similar  shape, 
switch  was  too  hard  to  reach). 

Pilot  comments: 

•  No  comments 


PV5.  Was  there  any  symbology  depicted  on  the  following  displays/pages  that  was  difficult 
to  quickly  and  easily  understand,  cluttered,  or  otherwise  difficult  to  use? 


Vertical  Situation  Display 

Yes  0% 

No  100% 

Horizontal  Situation  Display 

Yes  0% 

No  100% 

ESIS 

Yes  0% 

No  100% 

Engine  Instruments 

Yes  0% 

No  100% 

Digital  Map 

Yes  0% 

No  100% 

ASE 

Yes  0% 

No  100% 

NMS  Pages 

Yes  17% 

No  83% 

UAS 

Yes  17% 

No  83% 

If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  display/page,  b)  the 
symbology  that  was  difficult  to  understand,  c)  how  the  symbology  degraded  your 
performance,  and  d)  any  recommendations  you  have  for  improving  the  design  of  the  display 
page  and/or  symbology. 

Pilot  comments: 

•  If  possible,  could  the  current  radio  and  freq  be  a  selectable  "sticky"  on  every  page?  I 
know  some  pilots  don’t  mind  returning  to  the  coinmo  page  to  double  check  what  they  are 
on,  but  if  it  was  a  selectable  option  off  of  the  initial  page  to  turn  that  on,  I  would  use  it  to 
enhance  my  SA  and  reduce  my  workload  from  paging  back  to  the  coinmo  page. 
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•  The  NMS  and  UAS  pages  were  difficult  on  occasion  due  to  lack  of  experience. 

PV6.  How  easy  was  it  to  detect  the  following  indication  on  the  displays? 


Warning/Caution/Advisory  (MFD) 


- 1  Avg.  Rating  2 

1.5 

3 

4 

5 

Very 

*  Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Entry  into  Operational  Limits 


Avg.  Rating  2 

1.67 

3 

4 

5 

Very  ^ 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Low  Fuel 

Avg.  Rating 

2 

3 

4 

5 

1.33 

Very'' 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”,  list  which  indication  you  had 
difficulty  detecting/understanding,  why  you  had  difficulty,  and  any  recommendations 
to  make  the  indication  more  easily  detectable  and/or  understandable. 

Pilot  comments: 

•  No  comments. 

PV7.  Did  you  have  any  problems  using  the  overhead  panels  due  to  location,  inaccurate 
labeling,  etc? 

Yes  0%  No  50%  NU  50% 

PV8.  Did  you  have  any  problems  using  the  following  switches  and  controls  on  the 
instrument  panel? 

CMWS  Manual  Dispense  Switch  Yes  0%  No  0%  NU  100% 

SCAS  Control  Panel  Yes  0%  No  17%  NU  83% 


40 


FADEC  Switch 

Yes  0% 

No  0% 

NU  100% 

MFK 

Yes  0% 

No  87% 

NU  13% 

Armament  Control  Panel 

Yes  0% 

No  100% 

Channel  Select  Switch 

Yes  0% 

No  100% 

EBI  Filter  Bypass  Switch 

Yes  0% 

No  0% 

NU  100% 

If  yes,  list  the  switches  and/or  controls  and  describe  the  problem(s) 

Pilot  comments: 

•  No  comments. 

PV9a.  Did  you  have  any  problems  viewing  information  (symbology,  text,  etc)  on  the 


following  displays: 

6x8  Display 

Yes  33% 

No  67% 

5x7  Display 

Yes  0% 

No  100% 

ESIS  Display 

Yes  0% 

No  100% 

If  yes,  list  the  information  you  had  problems  viewing: 

Pilot  comments: 

•  Cipher  text  indication  (the  green  "C")  is  difficult  to  discern  on  the  white  background. 

•  6x8  HSD  should  incorporate  moving  compass  rose  such  as  5x7  displays  feature. 

PV9b.  Did  you  have  any  problems  using  the  bezels,  knobs  or  hot  keys  on  the  6x8 
display? 

Yes  0%  No  100% 

PV10.  Did  the  LPCAP  (digital  ICS)  restrict  cyclic  movement  in  the  right  seat? 

Yes  0%  No  100% 

If  yes,  describe  the  how  much  the  LPCAP  restricted  movement  and  in  which  axis: 

No  comments: 
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PV11.  Rate  whether  the  CDS5  software  was  much  more  or  less  effective  (quicker  and 
easier  to  use)  than  the  CDS4  software  you  have  used  on  the  KW. 


1 


Avg.  Rating 
1.5 


2 


Much  More"*  Somewhat 
Effective  More  Effective 


3 


About  The 
Same 


4 


Somewhat 
Less  Effective 


If  ‘Somewhat  Less’  or  ‘Much  Less  Effective’,  explain  why: 
Pilot  comments: 

•  No  comments. 


5 


Much  Less 
Effective 
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Appendix  E.  Top  Crewstation  Improvements 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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•  Occluded  compass  rose  overlay  on  the  6x8  map  page. 

•  When  the  pilot  drops  a  target  from  the  hover  BOB  UP  Switch,  a  box  identifying  the 
target  number  should  be  displayed  on  the  current  page  on  the  MFD  to  alert  the  pilot  the 
target  was  dropped. 

•  NMS  with  high  color  resolution,  laser  spot  tracking  capabilities,  visible  IR  laser  and  360 
degree  field  of  view. 

•  Compass  rose  displayed  on  all  map  pages  vs.  heading  tapes 

•  Once  again,  maybe  making  the  current  radio/freq  a  selectable  "sticky"  on  each  page  could 
reduce  workload  on  the  pilot  side. 

•  When  you  store  a  target  the  data  should  auto  fill  after  you  enter  the  grid.  I  am  not  sure  if 
this  was  a  simism  or  CDS5  specific. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


AAR 

after  action  review 

AOI 

area(s)  of  interest 

APEX 

Advanced  Prototype  Engineering  and  Experimentation 

ARH 

anned  reconnaissance  helicopter 

ARL 

U.S.  Army  Research  Laboratory 

ASL 

Applied  Science  Laboratories 

ATM 

Aircrew  Training  Manual 

BHIVE 

Battlefield  Highly  Immersive  Visual  Environment 

BIT 

Built-in  Test 

BWRS 

Bedford  Workload  Rating  Scale 

CAAS 

Common  Aviation  Architecture  System 

CDD 

Capability  Development  Document 

CDS 

control  display  system 

CIB 

Controlled  Image  Base 

CLSA 

China  Lake  Situational  Awareness 

CPC 

Comanche  Portable  Cockpit 

DAMA 

Demand  Assigned  Multiple  Access 

DIS 

Distributed  Interactive  Simulation 

DREN 

Defense  Research  and  Engineering  Network 

EDS 

Engineering  Development  Simulator 

EUD 

Early  User  Demonstration 

HFE 

Human  Factors  Engineering 

HLA 

High-Level  Architecture 

HOG 

Hands-On  Grip 
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HRED 

Human  Research  and  Engineering  Directorate 

HUMS 

Health  Usage  Monitoring  System 

IMC 

instrument  meteorological  conditions 

JVMF 

Joint  Variable  Message  Format 

KW  CASUP 

Kiowa  Warrior  Cockpit  and  Sensor  Upgrade  Program 

LEUE 

Limited  Early  User  Evaluation 

LOBL 

lock-on-before-launch 

LOI  2 

level  of  interoperability  2 

LUT 

Limited  User  Test 

MFD 

multifunctional  display 

MMS 

mast-mounted  sight 

NDI 

Northern  Digital  Incorporated 

NMS 

nose-mounted  sensor 

OneSAF 

One  Semi-Automated  Forces 

OTW 

out-the-window 

PCI 

Pilot-Crewstation  Interface 

PIM 

Pulse  Interval  Module 

SA 

situational  awareness 

SCAS 

Stability  Control  Augmentation  System 

SME 

subject  matter  expert 

SSDD 

Systems  Simulation  and  Development  Directorate 

SSQ 

Simulator  Sickness  Questionnaire 

TCMRA 

TRADOC  Capability  Manager,  Reconnaissance  Attack 

TRADOC 

Training  and  Doctrine  Command 

TS 

Total  Severity 

HP 

tactics,  techniques,  and  procedures 

UAS 

unmanned  aircraft  system 
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VFR 

VMC 

VTR 


visual  flight  rules 

visual  meteorological  conditions 

Video  Tape  Recorder 
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NO.  OF 

COPIES  ORGANIZATION 

1  DEFENSE  TECHNICAL 
(PDF)  INFORMATION  CTR 
DTIC  OCA 

1  DIRECTOR 

(PDF)  US  ARMY  RESEARCH  LAB 
IMAL  HRA 

1  DIRECTOR 

(PDF)  US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 

1  GOVT  PRINTG  OFC 
(PDF)  AMALHOTRA 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  C  A  DAVISON 
320  MANSCEN  LOOP  STE115 
FORT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  D 
T  DAVIS 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DG  J  RUBINSTEIN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  ARMC  FIELD  ELEMENT 

RDRL  HRM  CH  C  BURNS 
THIRD  A VE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  AWC  FIELD  ELEMENT 

RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BEL  VOIR  VA  22060-5828 


NO.  OF 

COPIES  ORGANIZATION 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  J  CHEN 
12423  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  CU  B  LUTAS-SPENCER 
6501  E  11  MILERD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AV  W  CULBERTSON 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  HUMAN  RSCH  AND  ENGRNG 

DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  C  CARSTENS 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DE  A  MARES 

1733  PLEASONTON  RD  BOX  3 
FORT  BLISS  TX  79916-6816 
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NO.  OF 

COPIES  ORGANIZATION 


8  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  SIMULATION  &  TRAINING 
TECHNOLOGY  CENTER 
RDRL  HRT  COL  M  CLARKE 
RDRL  HRT  I  MARTINEZ 
RDRL  HRT  T  R  SOTTILARE 
RDRL  HRT  B  N  FINKELSTEIN 
RDRL  HRT  G  A  RODRIGUEZ 
RDRL  HRT  I  J  HART 
RDRL  HRT  M  C  METEVIER 
RDRL  HRT  S  B  PETTIT 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  HQ  USASOC 

RDRL  HRM  CN  R  SPENCER 
BLDG  E2929  DESERT  STORM  DRIVE 
FORT  BRAGG  NC  28310 

1  ARMY  G1 
(PDF)  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

13  DIRUSARL 
(PDF)  RDRL  HR 

L  ALLENDER 
P  FRAN  ASZCZUK 
C  COSENZO 
RDRL  HRM 

P  SAVAGE -KNEPSHIELD 
RDRL  HRM  AL 
C  PAULILLO 
RDRL  HRM  B 
J  GRYNO  VICKI 
RDRL  HRM  C 
L  GARRETT 
RDRL  HRM  DJ 
D  DURBIN 
RDRL  HRS 
J  LOCKETT 
RDRL  HRS  B 
M  LAFIANDRA 
RDRL  HRS  C 
K  MCDOWELL 
RDRL  HRS  D 
B  AMREIN 
RDRL  HRS  E 
D  HEADLEY 
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Intentionally  Left  Blank. 
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